Mercedes-Benz Greener Manufacturing¶
Project 1
DESCRIPTION
Reduce the time a Mercedes-Benz spends on the test bench.
Problem Statement Scenario: Since the first automobile, the Benz Patent Motor Car in 1886, Mercedes-Benz has stood for important automotive innovations. These include the passenger safety cell with a crumple zone, the airbag, and intelligent assistance systems. Mercedes-Benz applies for nearly 2000 patents per year, making the brand the European leader among premium carmakers. Mercedes-Benz is the leader in the premium car industry. With a huge selection of features and options, customers can choose the customized Mercedes-Benz of their dreams.
To ensure the safety and reliability of every unique car configuration before they hit the road, the company’s engineers have developed a robust testing system. As one of the world’s biggest manufacturers of premium cars, safety and efficiency are paramount on Mercedes-Benz’s production lines. However, optimizing the speed of their testing system for many possible feature combinations is complex and time-consuming without a powerful algorithmic approach.
You are required to reduce the time that cars spend on the test bench. Others will work with a dataset representing different permutations of features in a Mercedes-Benz car to predict the time it takes to pass testing. Optimal algorithms will contribute to faster testing, resulting in lower carbon dioxide emissions without reducing Mercedes-Benz’s standards.
Following actions should be performed:
If for any column(s), the variance is equal to zero, then you need to remove those variable(s). Check for null and unique values for test and train sets. Apply label encoder. Perform dimensionality reduction. Predict your test_df values using XGBoost.
In [107]:
import os as os
os.getcwd()
Out[107]:
In [108]:
os.chdir('D:\Data Science\SL Downloads\dataset\Merc Benz')
In [8]:
os.getcwd()
Out[8]:
In [109]:
import pandas as pd
rawdata=pd.read_csv('./train/train.csv')
In [10]:
rawdata.head(5)
Out[10]:
In [11]:
# List of int, float,object columns
Types = rawdata.dtypes.reset_index()
Types.columns = ["Count", "Column Type"]
Types.groupby("Column Type").count()
Out[11]:
There are 8 categoric columns and remaining are numeric
In [12]:
rawdata.shape
Out[12]:
In [13]:
rawdata.describe()
Out[13]:
In [14]:
rawdata.isnull().sum()
Out[14]:
In [15]:
import seaborn as sns
sns.heatmap(rawdata.isnull())
Out[15]:
Looks like no null values in any columns
Checking unique values in each columns¶
In [128]:
#rawdata['X0'].unique()
for i in rawdata.drop(['ID','y'],axis=1):
print('Unique elements in.. '+ i +'-column')
print(rawdata[i].unique())
print('-----------------------------------------')
In [11]:
import matplotlib
In [129]:
import pandas_profiling as pp
pp.ProfileReport(rawdata)
Out[129]:
Observation¶
Number of observations 4209
Number of variables 378
BOOL 368 CAT 8 NUM 2
Null values- None
Target Y- Distinct count-2545
X11 and few others can be neglected, because all rows have same values. Many columns show high correlation
In [14]:
rawdata.head()
Out[14]:
In [16]:
## Annova check on Categorical columns
import statsmodels.api as sm
from statsmodels.formula.api import ols
col = ['X0', 'X1', 'X2', 'X3', 'X4', 'X5', 'X6', 'X8']
for i in col:
model = ols('y ~ ' + i , data=rawdata).fit()
print('Column : {}, F-statistic : {}, p-value : {}'.format(i, model.fvalue, model.f_pvalue))
High F statistics scores columns can be kept. X4 has pvalue close to 0.05, so we fail to reject Null Hypotghesis that- This columns dont affect target.
In [18]:
Types = rawdata.dtypes.reset_index()
Types.columns = ["Count", "Column Type"]
Types.groupby("Column Type").count()
# Numeric columns
numeric=Types[Types["Column Type"]=='int64'].Count
numeric
rawdata[numeric]
Out[18]:
F regression on Numeric columns¶
In [19]:
from sklearn.feature_selection import SelectKBest
from sklearn.feature_selection import f_regression
ftr = rawdata[numeric].drop(['ID'],axis=1)
trgt= rawdata.y
fs= SelectKBest(f_regression, k="all")
fs.fit(ftr,trgt)
Out[19]:
In [20]:
scores = list(fs.scores_)
pvalues = list(fs.pvalues_)
fcols= list(ftr.columns)
scores[0:5]
pvalues[0:5]
fcols[0:5]
Out[20]:
In [21]:
# List of tuples with feature and their importance
table = [(col, score, round (pvalue,4)) for col, score, pvalue in zip(fcols, scores, pvalues)]
print(table[0:5])
In [22]:
## sorting with pvalues in desc order of pvalues
table= sorted(table, key = lambda x: x[2], reverse = True)
print(table[0:5])
In [25]:
#Lets put them in a dataframe
newdf= pd.DataFrame(table, columns = ['Colname', 'fscore', 'pvalue'])
newdf
# There are some NAN values too
Out[25]:
In [26]:
# Our Null Hyp - These columns don't have any effect on target
# If pvalue < 0.05 we reject Null Hyp, otherwise fail to reject Null Hyp
# So for pvalues greater than 0.05 , they don't have any effect on target
# Hence, high p values can be rejected, meaning those columns can be neglected.
dropcols= newdf[newdf['pvalue'] > 0.05]
#dropcols=dropcols['Colname']
dropcols=dropcols.Colname.values
dropcols
Out[26]:
In [27]:
# Lets check Nan values
newdf.isnull().sum()
# 12 columns have Null values
newdf[newdf['fscore'].isnull()]
Out[27]:
In [28]:
rawdata['X11']
#rawdata['X11'].unique()
rawdata['X11'].sum()
## These columns all have only one value '0', also highlighted by Pandas profiling
## These columns can also be ignored.
Out[28]:
Now lets try Machine learning models for regression¶
- Without discarding any columns, lets check the performance
In [29]:
#One hot encoding
import pandas as pd
rawdata=pd.get_dummies(rawdata)
In [31]:
rawdata
Out[31]:
In [30]:
# Collecting X and Y
X = rawdata.drop(['ID','y'],axis=1).values
Y = rawdata['y'].values
In [200]:
X
Out[200]:
In [201]:
Y
Out[201]:
In [33]:
# Splitting X and y into training and testing sets
from sklearn.model_selection import train_test_split
X_train, X_test, Y_train, Y_test = train_test_split(X, Y, random_state=1, test_size=0.3)
In [34]:
print(X_train.shape)
print(Y_train.shape)
print(X_test.shape)
print(Y_test.shape)
Linear Regression¶
In [35]:
# import the ML algorithm
from sklearn.linear_model import LinearRegression
# instantiate
linreg = LinearRegression()
# fit the model to the training data (learn the coefficients)
linreg.fit(X_train, Y_train)
Out[35]:
In [191]:
linreg.coef_
Out[191]:
In [192]:
linreg.intercept_
Out[192]:
In [193]:
Y_test.shape
Out[193]:
In [204]:
#Making predictions
# make predictions on the testing set
Y_pred = linreg.predict(X_test)
In [196]:
Y_pred.shape
Out[196]:
In [205]:
# import libraries for metrics
import numpy as np
from sklearn import metrics
# Model evaluation metrics for regression
#print('y-intercept : ', linreg.intercept_)
#print('beta coefficients : ', linreg.coef_)
print('Mean Abs Error MAE : ', metrics.mean_absolute_error(Y_test, Y_pred))
print('Mean Sq Error MSE : ', metrics.mean_squared_error(Y_test, Y_pred))
print('Root Mean Sq Error RMSE : ', np.sqrt(metrics.mean_squared_error(Y_test, Y_pred)))
print('MAPE : ', np.mean(np.abs((Y_test - Y_pred) / Y_test)) * 100)
print('MPE : ', np.mean((Y_test - Y_pred) / Y_test) * 100)
print('r2 value : ', metrics.r2_score(Y_test, Y_pred))
Lets try KKN for regression¶
In [206]:
from sklearn import neighbors
In [207]:
# Modelling
clf = neighbors.KNeighborsRegressor()
clf.fit(X_train, Y_train)
Out[207]:
In [208]:
Y_pred=clf.predict(X_test)
In [209]:
Y_pred
Out[209]:
In [210]:
Y_test
Out[210]:
KNN metrics¶
In [211]:
# import libraries for metrics and reporting
import numpy as np
from sklearn import metrics
# Model evaluation metrics for regression
#print('y-intercept : ', linreg.intercept_)
#print('beta coefficients : ', linreg.coef_)
print('Mean Abs Error MAE : ', metrics.mean_absolute_error(Y_test, Y_pred))
print('Mean Sq Error MSE : ', metrics.mean_squared_error(Y_test, Y_pred))
print('Root Mean Sq Error RMSE : ', np.sqrt(metrics.mean_squared_error(Y_test, Y_pred)))
print('MAPE : ', np.mean(np.abs((Y_test - Y_pred) / Y_test)) * 100)
print('MPE : ', np.mean((Y_test - Y_pred) / Y_test) * 100)
print('r2 value : ', metrics.r2_score(Y_test, Y_pred))
Lets try Decision Tree for regression¶
In [217]:
from sklearn.tree import DecisionTreeRegressor
clf = DecisionTreeRegressor()
clf = clf.fit(X_train, Y_train)
In [218]:
Y_pred=clf.predict(X_test)
In [214]:
Y_pred
Out[214]:
In [215]:
Y_test
Out[215]:
Decision Tree metrics¶
In [219]:
# import libraries for metrics and reporting
import numpy as np
from sklearn import metrics
# Model evaluation metrics for regression
#print('y-intercept : ', linreg.intercept_)
#print('beta coefficients : ', linreg.coef_)
print('Mean Abs Error MAE : ', metrics.mean_absolute_error(Y_test, Y_pred))
print('Mean Sq Error MSE : ', metrics.mean_squared_error(Y_test, Y_pred))
print('Root Mean Sq Error RMSE : ', np.sqrt(metrics.mean_squared_error(Y_test, Y_pred)))
print('MAPE : ', np.mean(np.abs((Y_test - Y_pred) / Y_test)) * 100)
print('MPE : ', np.mean((Y_test - Y_pred) / Y_test) * 100)
print('r2 value : ', metrics.r2_score(Y_test, Y_pred))
Lets try Random Forest for regression¶
In [130]:
from sklearn.ensemble import RandomForestRegressor
clf = RandomForestRegressor()
clf = clf.fit(X_train, Y_train)
In [131]:
Y_pred=clf.predict(X_test)
In [232]:
Y_pred
Out[232]:
In [229]:
Y_test
Out[229]:
Random Forest metrics¶
In [233]:
# import libraries for metrics and reporting
import numpy as np
from sklearn import metrics
# Model evaluation metrics for regression
#print('y-intercept : ', linreg.intercept_)
#print('beta coefficients : ', linreg.coef_)
print('Mean Abs Error MAE : ', metrics.mean_absolute_error(Y_test, Y_pred))
print('Mean Sq Error MSE : ', metrics.mean_squared_error(Y_test, Y_pred))
print('Root Mean Sq Error RMSE : ', np.sqrt(metrics.mean_squared_error(Y_test, Y_pred)))
print('MAPE : ', np.mean(np.abs((Y_test - Y_pred) / Y_test)) * 100)
print('MPE : ', np.mean((Y_test - Y_pred) / Y_test) * 100)
print('r2 value : ', metrics.r2_score(Y_test, Y_pred))
Some tunning on Random Forest¶
With K-Folds cross-validator
Provides train/test indices to split data in train/test sets. Split dataset into k consecutive folds (without shuffling by default).Each fold is then used once as a validation while the k - 1 remaining folds form the training set.
In [3]:
from sklearn.model_selection import KFold
KF= KFold(n_splits=10)
Parameters of RF regressor
- n_estimators int, default=100 -The number of trees in the forest.
- min_samples_split, int or float, default=2 -- The minimum number of samples required to split an internal node:
- max_depth, int, default=None, The maximum depth of the tree. If None, then nodes are expanded until all leaves are pure or until all leaves contain less than min_samples_split samples.
- criterion {“mse”, “mae”}, default=”mse” The function to measure the quality of a split. Supported criteria are “mse” for the mean squared error, which is equal to variance reduction as feature selection criterion, and “mae” for the mean absolute error.
- max_features{“auto”, “sqrt”, “log2”}, int or float, default=”auto” The number of features to consider when looking for the best split:
In [132]:
clf = RandomForestRegressor(n_estimators=50,min_samples_split=0.1, max_depth=10, criterion='mse', max_features='sqrt')
clf = clf.fit(X_train, Y_train)
Evaluate a score by cross-validation
- estimator= model to fit, here we are using RF
- cv,int, cross-validation generator or an iterable, default=None. Determines the cross-validation splitting strategy. We will use 10 from KF 3.scoring-str or callable, default=None A str (see model evaluation documentation) or a scorer callable object / function with signature scorer(estimator, X, y) which should return only a single value. Similar to cross_validate but only a single metric is permitted.
In [259]:
from sklearn.model_selection import cross_val_score
KFresult=cross_val_score(estimator=clf,X=X,y=Y,cv=KF, scoring='neg_mean_squared_error')
print( 'Mean Squared Error : ', KFresult.mean())
Feature elimination¶
In [295]:
# from F regression on numeric cols
dropcols
# columns with only one value in them
dropcols2=newdf[newdf['fscore'].isnull()].Colname.values
# from Annova on Cat columns- X4 can be discarded
In [307]:
rawdata=pd.read_csv('./train/train.csv')
# dropping x4
rawdata = rawdata.drop(['X4'],axis=1)
#dropping dropcols columns
dropcols=pd.Series(list(dropcols))
rawdata=rawdata.drop(rawdata[dropcols],axis=1)
#dropping dropcols2 columns
dropcols=pd.Series(list(dropcols2))
rawdata=rawdata.drop(rawdata[dropcols2],axis=1)
Re-trying Machine Learning again after feature elimination¶
In [310]:
#One hot encoding
import pandas as pd
rawdata=pd.get_dummies(rawdata)
rawdata
Out[310]:
In [311]:
# Collecting X and Y
X = rawdata.drop(['ID','y'],axis=1).values
Y = rawdata['y'].values
In [313]:
Y
Out[313]:
In [46]:
# Splitting X and y into training and testing sets
from sklearn.model_selection import train_test_split
X_train, X_test, Y_train, Y_test = train_test_split(X, Y, random_state=1, test_size=0.3)
print(X_train.shape)
print(Y_train.shape)
print(X_test.shape)
print(Y_test.shape)
In [66]:
## KKN for regression
from sklearn import neighbors
# Modelling
clf = neighbors.KNeighborsRegressor(n_neighbors=17, metric='hamming', weights= 'distance')
clf.fit(X_train, Y_train)
Y_pred=clf.predict(X_test)
Params for KNN
1.n_neighborsint, default=5. Number of neighbors to use by default for kneighbors queries.
2.weights{‘uniform’, ‘distance’} or callable, default=’uniform’ weight function used in prediction. Uniform weights are used by default.
3.algorithm {‘auto’, ‘ball_tree’, ‘kd_tree’, ‘brute’}, default=’auto’. Algorithm used to compute the nearest neighbors.
- metric-str or callable, default=’minkowski’. The distance metric to use for the tree. The default metric is minkowski, and with p=2 is equivalent to the standard Euclidean metric.
In [67]:
## KNN metrics
# import libraries for metrics and reporting
import numpy as np
from sklearn import metrics
# Model evaluation metrics for regression
#print('y-intercept : ', linreg.intercept_)
#print('beta coefficients : ', linreg.coef_)
print('Mean Abs Error MAE : ', metrics.mean_absolute_error(Y_test, Y_pred))
print('Mean Sq Error MSE : ', metrics.mean_squared_error(Y_test, Y_pred))
print('Root Mean Sq Error RMSE : ', np.sqrt(metrics.mean_squared_error(Y_test, Y_pred)))
print('MAPE : ', np.mean(np.abs((Y_test - Y_pred) / Y_test)) * 100)
print('MPE : ', np.mean((Y_test - Y_pred) / Y_test) * 100)
print('r2 value : ', metrics.r2_score(Y_test, Y_pred))
With Kfold
In [68]:
from sklearn.model_selection import cross_val_score
KFresult=cross_val_score(estimator=clf,X=X,y=Y,cv=KF, scoring='neg_mean_squared_error')
print( 'Mean Squared Error : ', KFresult.mean())
In [69]:
## Lets try Decision Tree for regression
from sklearn.tree import DecisionTreeRegressor
clf = DecisionTreeRegressor()
clf = clf.fit(X_train, Y_train)
Y_pred=clf.predict(X_test)
In [70]:
## Decision Tree metrics
# import libraries for metrics and reporting
import numpy as np
from sklearn import metrics
# Model evaluation metrics for regression
#print('y-intercept : ', linreg.intercept_)
#print('beta coefficients : ', linreg.coef_)
print('Mean Abs Error MAE : ', metrics.mean_absolute_error(Y_test, Y_pred))
print('Mean Sq Error MSE : ', metrics.mean_squared_error(Y_test, Y_pred))
print('Root Mean Sq Error RMSE : ', np.sqrt(metrics.mean_squared_error(Y_test, Y_pred)))
print('MAPE : ', np.mean(np.abs((Y_test - Y_pred) / Y_test)) * 100)
print('MPE : ', np.mean((Y_test - Y_pred) / Y_test) * 100)
print('r2 value : ', metrics.r2_score(Y_test, Y_pred))
In [71]:
## Lets try Random Forest for regression
from sklearn.ensemble import RandomForestRegressor
clf = RandomForestRegressor()
clf = clf.fit(X_train, Y_train)
Y_pred=clf.predict(X_test)
In [ ]:
## Random Forest metrics
# import libraries for metrics and reporting
import numpy as np
from sklearn import metrics
# Model evaluation metrics for regression
#print('y-intercept : ', linreg.intercept_)
#print('beta coefficients : ', linreg.coef_)
print('Mean Abs Error MAE : ', metrics.mean_absolute_error(Y_test, Y_pred))
print('Mean Sq Error MSE : ', metrics.mean_squared_error(Y_test, Y_pred))
print('Root Mean Sq Error RMSE : ', np.sqrt(metrics.mean_squared_error(Y_test, Y_pred)))
print('MAPE : ', np.mean(np.abs((Y_test - Y_pred) / Y_test)) * 100)
print('MPE : ', np.mean((Y_test - Y_pred) / Y_test) * 100)
print('r2 value : ', metrics.r2_score(Y_test, Y_pred))
In [41]:
## tunning on Random Forest
from sklearn.model_selection import KFold
KF= KFold(n_splits=10)
clf = RandomForestRegressor(n_estimators=50,min_samples_split=0.1, max_depth=10, criterion='mse', max_features='sqrt')
clf = clf.fit(X_train, Y_train)
from sklearn.model_selection import cross_val_score
KFresult=cross_val_score(estimator=clf,X=X,y=Y,cv=KF, scoring='neg_mean_squared_error')
print( 'Mean Squared Error : ', KFresult.mean())
Trying Gradient Boosting¶
Params¶
- n_estimatorsint, default=100, The number of boosting stages to perform. Gradient boosting is fairly robust to over-fitting so a large number usually results in better performance.
- min_samples_splitint or float, default=2.The minimum number of samples required to split an internal node:
- learning_ratefloat, default=0.1. learning rate shrinks the contribution of each tree by learning_rate. There is a trade-off between learning_rate and n_estimators.
- loss{‘ls’, ‘lad’, ‘huber’, ‘quantile’}, default=’ls’. loss function to be optimized. ‘ls’ refers to least squares regression. ‘lad’ (least absolute deviation) is a highly robust loss function solely based on order information of the input variables. ‘huber’ is a combination of the two. ‘quantile’ allows quantile regression
- criterion{‘friedman_mse’, ‘mse’, ‘mae’}, default=’friedman_mse’. The function to measure the quality of a split. Supported criteria are “friedman_mse” for the mean squared error with improvement score by Friedman, “mse” for mean squared error, and “mae” for the mean absolute error. The default value of “friedman_mse” is generally the best as it can provide a better approximation in some cases.
In [31]:
# Fit regression model
from sklearn.ensemble import GradientBoostingRegressor
params = {'n_estimators': 1500,
'max_depth': 4,
'min_samples_split': 2,
'learning_rate': 0.005,
'loss': 'ls'}
gbr = GradientBoostingRegressor(**params)
In [32]:
# Train GB regressor
from sklearn.model_selection import train_test_split
X_train, X_test, Y_train, Y_test = train_test_split(X, Y, random_state=1, test_size=0.3)
gbr.fit(X_train, Y_train)
Out[32]:
In [45]:
from sklearn.model_selection import cross_val_score
KFresult=cross_val_score(estimator=gbr,X=X,y=Y,cv=KF, scoring='neg_mean_squared_error')
print( 'Mean Squared Error : ', KFresult.mean())
In [ ]:
# Tuning Gradient Boost
from sklearn.model_selection import cross_validate
from sklearn.model_selection import KFold
KF=KFold(n_splits=5, random_state=20)
scor={'r2':'r2', 'MSE':'neg_mean_squared_error'}
scores=cross_validate(estimator=gbr,X=X_train,y=Y_train,cv=KF,scoring=scor,return_train_score=True)
#scores.keys()
In [46]:
print('Train MSE')
print(scores['train_MSE'].mean())
print('Train R2')
print(scores['train_r2'].mean())
print('-------------vs---------------')
print('Test MSE')
print(scores['test_MSE'].mean())
print('Test R2')
print(scores['test_r2'].mean())
Trying XG Boost¶
In [47]:
pip install xgboost
In [72]:
# train test split
X_train, X_test, Y_train, Y_test = train_test_split(X,
Y,
test_size=0.2,
random_state=123)
In [87]:
X_test
Out[87]:
In [73]:
import xgboost as xgb
train = xgb.DMatrix(X_train,Y_train)
test = xgb.DMatrix(X_test, Y_test)
In [97]:
# parameters for tuning
params={'max_depth': 7,
'min_child_weight': 2,
'eta': 0.005,
'subsample': 0.8,
'colsample_bytree': 1,
'objective': 'reg:linear',
'eval_metric': 'mae'}
In [98]:
num_boost_round = 999
In [100]:
%%time
model = xgb.train(
params,
train,
num_boost_round=num_boost_round,
evals=[(test, "Test")],
early_stopping_rounds=10
)
In [101]:
# Predict
from sklearn import metrics
Y_pred = model.predict(train)
print("Training : metrics ...")
print('Mean Abs Error MAE : ', metrics.mean_absolute_error(Y_train, Y_pred))
print('Mean Sq Error MSE : ', metrics.mean_squared_error(Y_train, Y_pred))
print('Root Mean Sq Error RMSE : ', np.sqrt(metrics.mean_squared_error(Y_train, Y_pred)))
print('r2 value : ', metrics.r2_score(Y_train, Y_pred))
Y_pred = model.predict(test)
print('\n')
print("Testing : metrics ...")
print('Mean Abs Error MAE : ', metrics.mean_absolute_error(Y_test, Y_pred))
print('Mean Sq Error MSE : ', metrics.mean_squared_error(Y_test, Y_pred))
print('Root Mean Sq Error RMSE : ', np.sqrt(metrics.mean_squared_error(Y_test, Y_pred)))
print('r2 value : ', metrics.r2_score(Y_test, Y_pred))
R2 of 0.64 & MSE of 58 on Training vs 0.61 R2 & 58 MSE on Testing looks okay. Increasing the max depth to a higher number (eg:20) gives better results on Testing set, but training score is poor. At max depth of 7, and other tuning parameters, this trade off gives a decent result.
In [ ]: